Skip to content

Autopilot wrapp + a gate correctness fix - #11

Open
sameeeeeeep wants to merge 5 commits into
mainfrom
feat/autopilot-wrapp
Open

Autopilot wrapp + a gate correctness fix#11
sameeeeeeep wants to merge 5 commits into
mainfrom
feat/autopilot-wrapp

Conversation

@sameeeeeeep

Copy link
Copy Markdown
Owner

The headline is not the wrapp

Chasing Autopilot's token surface turned up a correctness bug in the gate. The claude-code backend counted only usage.input_tokens and ignored cache_creation_input_tokens / cache_read_input_tokens. Measured on a real call: 451 reported, 34,585 actually consumed.

server.ts feeds that straight into gate.recordCompletion(), so every per-origin tokens/day budget was ~77x more permissive than the consent card promised. Budgets are an advertised protection; they were not enforcing what they claimed. Default raised 200K → 2M/day to match honest counting (one decision-sized call is ~26K, so the old default was ~7 calls/day).

Autopilot

The operating cockpit: lend it a brand / project / idea and it drafts the whole slate — voice, ad angle, channel, next move — with zero input. Choosing ripples downstream. Moves are staged auto | approve | manual, a taxonomy taken from brandbrain's OS as a concept, not code.

Resolved first: separate wrapp, not a third brandbrain studio. The roadmap's "three verbs" argument was retro-justification — the separation had entered the record as a scope guard, and the studio option was never evaluated. It was: brandbrain's studio engine is finite by construction (seq.every(id => locks[id])), the cockpit is exactly the part that does not generalise (ideabrain already can't reach the OS), and the OS is one-brand-at-a-time. Full reasoning in ROADMAP.md.

contextKinds is ["brand","project","idea"], not ["brand"] — the narrow version would have broken the idea→run graduation.

The catalog-wide bug this shook out

Making the harness storage stub real (it discarded every write) turned reel/marquee/arcade red immediately. The double-onReady race was live in 21 wrapps, invisible because the fixture lied. All 21 + the template now carry hydrated.

When a fixture has to lie to keep the suite green, the bug is in the product.

Pricing made consistent

The page promised "no second tier" while TOKENS.md documents a Pro paying developers 75%. Rewritten to CLOUD.md's stated line — your infra is free, our cloud is Pro; we charge for infrastructure we run, never for AI access. That rule then contradicted 12 catalog entries selling power-user batch features as Pro; all 13 are now one shape.

The hero mock also showed three moves the product cannot generate. It now uses movesFor()'s actual output.

Verification

  • Harness 70/70 with real storage (was 68 cells / 64 pass)
  • examples/apps/proof/run-autopilot-storage.mjs — 19/19 against a real daemon on a throwaway RELAY_DIR: one company = one file, editing A never rewrites B, torn records skipped, traversal ids refused, per-origin isolation
  • Redline's real audit run on the landing page; safe findings applied

Known gaps

  • tokens: 420_000 in the catalog is still dev-reported (it's build cost, not runtime).
  • Redline is a copy editor, not a truth-checker — it caught neither the pricing contradiction nor the fabricated mock, because it only sees page HTML, never the repo. Grounding its audit in the lent project context is the obvious next upgrade.

🤖 Generated with Claude Code

sameeeeeeep and others added 5 commits July 26, 2026 04:04
… fix

Autopilot ships as a standalone wrapp: lend it a brand/project/idea and it
drafts the operating slate cold — voice, ad angle, channel, next move. Choosing
ripples downstream; nothing irreversible happens without you.

Resolved first: separate wrapp, NOT a third brandbrain studio. The roadmap's
"three verbs" argument was retro-justification — the separation had entered the
record as a scope guard, and a studio was never evaluated. It was. brandbrain's
studio engine is finite by construction (seq.every(id => locks[id])) with no
concept of a recurring decision; the cockpit is the part that does NOT
generalise (compileTasks is a hand-written if-chain, and the !active.path gate
already stops ideabrain reaching the OS); and the OS is one-brand-at-a-time,
which contradicts a portfolio at the state level.

contextKinds is ["brand","project","idea"], not ["brand"] — the latter re-narrows
a wrapp meant to generalise past D2C and breaks the idea→run graduation.

Also took brandbrain's auto|approve|manual taxonomy as a concept, not code. It
is what makes "autonomous" honest.

THE BIGGER RESULT — a live race in 21 wrapps, previously masked by the harness.
onReady fires twice (mountConnect + the returning-user probe). With storage that
really returns what you wrote, the second pass re-reads the run the first just
saved and REPLACES the in-memory object the running pipeline holds, so the
pipeline completes into a detached orphan and the UI sits on a run that never
finishes. The harness had been hiding this since 2026-07-18 by discarding every
write. Backing the mock with a real per-page store turned reel/marquee/arcade
red immediately. Fixed in all 21 wrapps and in the wrapp template so new ones
inherit it (cartridge.js already had it; it was never propagated).

Verified:
- harness 70/70 (was 68 runs → 64/2/2), now WITH real storage
- examples/apps/proof/run-autopilot-storage.mjs — 19/19 against a real daemon:
  one company = one file, editing A never rewrites B, torn records skipped,
  traversal ids refused, per-origin isolation (npm run try-autopilot)
- interactions driven end-to-end: cold open, choose, 2-level ripple, unchoose,
  take-back-inherited, escape hatch, multi-company, budget tiers, error +
  recovery, responsive at 375/1440

Token figure in the catalog stays dev-reported — live measurement needs Claude
Code signed in on this machine.

Co-Authored-By: Claude <noreply@anthropic.com>
The claude-code backend read only `usage.input_tokens`, ignoring
`cache_creation_input_tokens` and `cache_read_input_tokens`. Measured on a
real call: 451 reported vs 34,585 actually consumed.

That number is not cosmetic. server.ts feeds it straight into
`gate.recordCompletion()`, so every per-origin tokens/day budget was
enforcing ~77x more than the consent card promised — and every wrapp surface
showing "tokens spent" understated by two orders of magnitude.

Cached input is still input: it is context the model processed and the
account was billed for. The OpenAI-shaped backends were already correct
(`prompt_tokens` is a total).

Paired with the correction, the default budget goes 200K -> 2M/day. One
decision-sized call measures ~26K tokens, so the old default was ~7 calls a
day; a wrapp that cold-opens would have burned a third of it before the user
typed anything. The gate is meant to stop runaway origins, not ordinary use.

Co-Authored-By: Claude <noreply@anthropic.com>
Pricing. The Autopilot page promised "Free, and there is no second tier"
while TOKENS.md documents a Pro subscription paying developers 75%. Rewritten
to the line CLOUD.md already states: your infra is free, our cloud is Pro —
we charge for infrastructure we run, never for AI access. Team Mode over your
own LAN or a self-hosted relay stays free; the hosted relay and encrypted
backup are Pro.

That rule then contradicted the catalogue, where 12 wrapps advertised a
different Pro made of power-user batch features ("batch shot lists",
"publish to a domain", "whole-site crawls") that cost us nothing to run and
which the stated principle makes free. All 13 pro lists are now the same
shape: a shared board plus encrypted backup.

Honesty. The hero mock showed three staged moves the product cannot generate
(outreach to 32 outfitters, calling a co-packer). movesFor() emits exactly
three shapes; the mock now uses them. Fabricating capability on a page whose
pitch is "revenue would need a connected store, so it says not connected"
undercuts the only thing being sold.

Redline pass (its real audit, run on the page): deduplicated the hero /
feature-01 / free-bullet triple, replaced the unbacked "Your data stays
yours" chip with two claims that are backed, and cut "Free core, forever"
from 4 uses to 1. Redline's "glossary card in a pricing slot" finding is
resolved by the slot now holding an actual plan.

Co-Authored-By: Claude <noreply@anthropic.com>
The harness only ever said yes. grant() approved everything asked for, health
was always green, and every tool call succeeded — so the entire class of "the
user declined" or "the daemon is down" was untested across the catalogue,
despite the wrapp skill explicitly warning that a wrapp must "read the
returned grant and handle partial approval".

Adds ?refuse=<mode>, opt-in per run so the default sweep is byte-identical:
  partial   only the first tool and first model survive the consent card
  tools     every tool unticked, models still granted
  connect   the user closes the card
  asleep    daemon installed but unreachable — fails at the TRANSPORT, not
            just in the health probe (flipping health alone let wrapps sail
            straight past it; Autopilot drafted a full slate against a daemon
            reporting "not running", which tested nothing)
  toolcall  full grant, but every tool INVOCATION fails at run time

Swept all 34 wrapps under `asleep`: 34/34 degrade cleanly — zero errors, no
hangs, every one falls back to a sensible pre-connect state. No bug found,
which is the point of running it.

Recorded while doing this: `partial` and `tools` cannot yet find anything,
because the harness's baseline grant is built from an EMPTY scope
(`GRANT = grant({})`) and the returning-user path never calls connect. So
every wrapp already runs with zero granted tools and the full default model
list, and the mock's stream ignores grants entirely. Making partial-approval
testable means seeding the grant from each wrapp's declared scope first —
a bigger change than a flag, and the real blocker.

Co-Authored-By: Claude <noreply@anthropic.com>
Cheap 80% of "seed real grants in the harness", which was the expensive fix I
reached for first. The harness cannot see a scope mistake at all: its grant is
fabricated from an empty scope and the responder ignores grants entirely. Rather
than re-architect that, read each wrapp's own source and compare what it CALLS
against what it DECLARES.

Catches the two documented gotchas: a tool used but not granted (Higgsfield's
whole-connector wildcard, WebSearch, WebFetch), and a model called that the scope
does not list (grants are exact-match).

Deliberately conservative. genImage() defined-but-never-called is template
plumbing, not a finding — several wrapps carry the helper unused, and counting
those would have produced a false positive on arcade, which is exactly what a
cruder grep did before this was written.

Result on the current catalogue: 42 sources, zero findings — every wrapp already
declares what it calls. Verified non-vacuous by injecting both failure kinds into
autopilot and confirming both fire.

This is a "you will be denied in production" check, never a security claim: the
gate is default-deny and enforced daemon-side, so these already fail loudly on a
developer's first real run. The lint just moves discovery to build time.

Not chained into `npm run build` on purpose — a lint that can abort a 36-entry
esbuild run is how builds get bypassed. Run it with `npm run lint:scope`.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant